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SUBGROUP OF A NILPOTENT-BY-ABELIAN GROUP 



GERARD ENDIMIONI 

Abstract. Among other things, we prove that the group of au- 
tomorphisms fixing every normal subgroup of a (nilpotent of class 
c)-by-abelian group is (nilpotent of class < c)-by-metabelian. In 
particular, the group of automorphisms fixing every normal sub- 
group of a metabelian group is soluble of derived length at most 3. 
An example shows that this bound cannot be improved. 



1. Introduction and results. 

If G is a group, we write Aut(G) for the group of all automorphisms 
of G. An automorphism (p G Aut(G) is said to be normalii ^p{H) = H 
for each normal subgroup H of G. These automorphisms form a nor- 
mal subgroup of Aut(G), denoted by Autn(G). Obviously, Autn(G) 
contains the subgroup Inn(G) of all inner automorphisms of G. These 
subgroups can coincide, for instance when G is a nonabelian free group 
[5]. We have a similar result when G is a nonabelian free nilpotent 
group of class c = 2, but if c > 3, then the quotient Autn(G)/Inn(G) is 
infinite [2] . Also one can find in [3j a description of the normal automor- 
phisms of a free metabelian nilpotent group. In general, the structure 
of Autn(G) can be quite diverse since for an arbitrary finite group F, 
there is a finite semisimple group G such that Autn(G)/Inn(G) has a 
subgroup isomorphic with F |7|. Nevertheless, as one can expect, it is 
possible to obtain some information about Autn(G) when G belongs to 
certain classes of groups. For instance, when G is nilpotent, the group 
Autn(G) is nilpotent-by-abelian [4]. In the same paper, the authors 
show that Autn(G) is polycyclic when G is. In particular, if G is a 
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finite soluble group, then so is Autn(G). We do not know if this asser- 
tion remains true without the term "finite" . In other words, is Autn(G) 
soluble when G is soluble ? Here we respond to this question in the 
positive when G is soluble of derived length < 2, and more generally 
when G is nilpotent-by-abelian. 

Theorem 1. The group of all normal automorphisms of a (nilpotent 
of class c)-by-abelian group is (nilpotent of class < c)-by-metabelian. 

When c = 1, we obtain: 

Corollary 1. The group of all normal automorphisms of a metabelian 
group is soluble of derived length at most 3. 

As usual, denote by 5*4 the symmetric group of degree 4 and by its 
alternating subgroup. Then Autn(A4) = Aut(A4) is isomorphic to S^. 
Since is metabelian and ^4 is soluble of derived length 3, the bound 
of the derived length given in the corollary above cannot be improved. 

We shall see that the proof of Theorem 1 also leads to the following 
result. 

Theorem 2. Let G be a (nilpotent of class c)-by-abelian group. Sup- 
pose that its abelianization G/G' is either finite or infinite non-periodic 
(that is the case for example when G is finitely generated). Then the 
group of all normal automorphisms of G is virtually ( nilpotent of class 
< c)-by-abelian. 

In particular, the group of normal automorphisms of a finitely gen- 
erated metabelian group is virtually metabelian. 

Since a supersoluble group is nilpotent-by-abelian and finitely gener- 
ated, it follows from Theorem 2 that its group of all normal automor- 
phisms is virtually nilpotent-by-abelian. In fact, we have a stronger 
result. 

Theorem 3. The group of all normal automorphisms of a supersoluble 
group is finitely generated and nilpotent-by- (finite and supersoluble) . 
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2. Proofs. 



Let G be a group and let G' denote its derived subgroup. Clearly, 
each normal automorphism / G Autn(G) induces in G/G' a normal au- 
tomorphism /' G Autn(G/G'). Consider the homomorphism $ : Autn(G) 
Autn(G/G') defined by $(/) = /' and put K = ker $. In other words, 
K is the set of normal lA-automorphisms of G (recall that an automor- 
phism of G is said to be an I A- automorphism if it induces the identity 
automorphism in G/G'). Before to prove Theorem 1, we establish a 
preliminary result regarding the elements of K. 

Lemma 1. Let k he a positive integer. In a group G, consider an 
element a G 7fc(G'), where 'yk{G') denotes the kth term of the lower 
central series of G' . If f and g are normal I A- automorphisms of G 
(i.e. f,g E K), we have: 

(i) for all u & G, f{u''^au) = u~^f{a)u mod 7^+1 (G'); 

(ii) of'^ogo f{a) = a mod 7^+1 (G'). 

Proof, (i) We have f{u) = uu' for some u' G G', whence 

f{u~^au) = u'^^u~^ f{a)uu' = f {a)u[u~^ f {a)u, u'], 

where the commutator [x,y] is defined by [x,y] = x~^y~^xy. Since 
[u'^f{a)u,u'] belongs to 7fc+i(G'), the result follows. 

(ii) Since / and g preserve the normal closure of a in G, there exist 
elements Ui, . . . ,Ur,Vi, . . . ,Vs G G and integers Ai, . . . , A^, Hi, . . . , Hs 
such that 



r 



s 



/(«) = n 



a ^Ui and g{a) = 
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Notice that in these products, the order of the factors is of no conse- 
quence modulo 7fc+i(G'). Using (i), one can then write 

r r 

i=l 1=1 
r s 

i=l j=l 

r s 

= ]^]^[t;j,Mj]"^w7^Mria^'''%jt;j[t;j-,Mj] mod 7fc+i (G") • 
1=1 j=i 

Since 

[vj,Ui\'^vJ^ur^a^'''^UiVj[vj,Ui\ = vJ^u^^a^'^^UiVj mod 7^+1(6"), 
we obtain 

r s 

9 o f{o) = WWvj\i^a^^^'^UiVj = fo g{a) mod 7^+1(6"). 
i=i j=i 

It follows g^^ o /^^ o g o f{(i) = a mod 7^+1(6"), which is the desired 
result. □ 

Proof of Theorem 1. Thus we suppose now that G is (nilpotent of class 
c)-by-abelian, and so 7c+i(G") is trivial. Recall that K is the kernel 
of the the homomorphism $: Autn(G) Autn(G/G') defined above. 
The group G/G' being abelian, each normal automorphism of G/G' is 
in fact a power automorphism, that is, an automorphism fixing setwise 
every subgroup of G/ G' . Since the group of all power automorphisms of 
a group is always abelian pQ, so is Autn(G/G'). The group Autn(G)/K 
being isomorphic with a subgroup of Autn(G/G'), it is abelian, and so 
Autn(G)/K' is metabelian. It remains to see that K' is nilpotent, of 
class at most c. For that, we notice that K' stabilizes the normal series 
(of length c + 1) 

1 = 7c+i(G") < 7c(G") < ■ ■ -72(^0 < 7i(G") = G" < G. 

Indeed, the induced action of K' on the factor G/G' is trivial since 
every element of K is an lA-automorphism. On the other factors, it 
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is a consequence of the second part of the lemma above. By a well- 
known result of Kaluznin (see for instance [HI P- 9]), it follows that K' 
is nilpotent of class < c, as required. □ 

Proof of Theorem 2. First notice that in an abelian group which is ei- 
ther finite or infinite non-periodic, the group of all power automor- 
phisms is finite. That is trivial when the group is finite. In the second 
case, the group of power automorphisms has order 2, the only non- 
identity power automorphism being the inverse function x ^ (see 
for instance [1, Corollary 4.2.3] or [6, 13.4.3]). Therefore, coming back 
to the proof of Theorem 1 when G/G' is either finite or infinite non- 
periodic, we can assert that Autn(G)/K is finite. Since K is (nilpotent 
of class < c)-by-abelian, the result follows. □ 

Proof of Theorem 3. Let G be a supersoluble group. Since G is poly- 
cychc, so is Autn(G) [4]. Thus Autn(G) is finitely generated. Let us 
prove now that Autn(G) is nilpotent-by- (finite and supersoluble). By 
a result of Zappa [61 5.4.8], there is a normal series 

(1) 1 = G„ < < ■ ■ ■ < Gi < Go = G 

in which each factor is cyclic of prime or infinite order. For any k G 
{0,l,...,m}, we denote by the set of all normal automorphisms of 
G which stabilize the series Gk ^ Gk-i < ■ ■ ■ < Gi < Go = G (we put 
Fo = Autn(G)). Clearly, Fo, Fi, . . . , F^ forms a decreasing sequence of 
normal subgroups of Autn(G). Using once again the result of Kaluznin 
[Hp. 9], we can assert that F.^ is nilpotent (of class at most m — 
1) since F^ stabilizes the series ([T]) above. It remains to prove that 
Autn(G)/Fm is supersoluble and finite. For each integer k (with < 
/c < m — 1), consider the homomorphism ^^'-^k Aut(Gk/Gk+i), 
where for any / G F^, '^k{f) is defined as the automorphism induced 
by / on Gk/Gk+i- We observe that Aut(Gk/Gk+i) is finite cychc and 
that ker\I'fc = F^+i. Consequently, the factor Ffc/F^+i is finite cychc, 
since it is isomorphic to a subgroup of Aut(Gk/Gk+i). It follows that 

1 = (F„/F„) < (F^_i/F„) <■■■< (Fi/F„) < (Fo/F„) = (Aut,(G)/F^) 

forms a normal series in which each factor is finite cyclic. Thus Autn(G) /F 
is supersoluble and finite and the proof is complete. □ 
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